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ABSTRACT 


Government  leaders’  policy  decisions  can  significantly  impact  the  acquisition  of  weapon  systems  for  warfighters. 
Two  such  decisions  were  made  in  1998  and  2004  within  the  U.S.  Army.  While  these  decisions  may  impact 
system  acquisition,  they  can  also  affect  important  sub-elements  of  the  acquisition  process,  such  as  sustainment 
support  of  software  operating  within  systems.  This  study  researched  software  sustainment  support  within  the 
Army  Command,  Control,  Communications,  Computer,  Intelligence,  Sustainment,  and  Reconnaissance  (C4ISR) 
community  following  enactment  of  these  two  acquisition  policy  decisions.  Specifically,  the  study  researched  the 
perception  of  user/operators,  Program  Executive  Officer/Program  Manager  (PEO/PM)  and  software  sustainment 
support  activity  (SSSA)  personnel  about  the  quality,  cost,  timeliness,  and  thoroughness  of  software  support  before 
and  after  these  two  policy  decisions.  The  study  also  assessed  differences  in  the  perception  of  software  support 
within  different  sectors  of  the  C4ISR  community. 

Constrained  by  limited  response  to  the  research  survey,  the  study  results  were  based  on  a  very  small  sample  of  the 
Army  C4ISR  community.  The  study  was  also  unable  to  control  other  factors  that  could  have  influenced  perception 
of  software  sustainment  support.  Within  these  constraints,  the  study  indicates  that,  in  general,  perceptions  of 
software  sustainment  support  are  the  same  or  have  only  slightly  improved  since  the  two  policy  decisions.  However, 
various  sectors  of  the  Army  C4ISR  community  have  different  ideas  about  which  aspects  of  software  sustainment 
support  should  be  further  improved.  While  most  PEO/PM  personnel  believe  improvements  are  required  in  areas 
of  the  timeliness  of  software  upgrades  and  system  accreditation,  most  SSSA  personnel  believe  improvements  are 
required  in  the  cost  of  field  software  support. 
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CHAPTER  1 

INTRODUCTION 


Prior  to  1998,  life-cycle  support  for  U.S.  Army  weapon  systems  transitioned  from  Program  Management 
Offices  (PMO)  that  developed  systems  to  Army  Materiel  Command  (AMC)  major  subordinate  commands 
(MSCs)  providing  logistical  support  until  end  of  system  life.  In  an  attempt  to  get  PMs  and  MSCs  to  work  together 
more  closely,  Gil  Decker,  former  Assistant  Secretary  of  the  Army  for  Research,  Development,  and  Acquisition 
(ASA[RDA])  (Decker,  1997),  directed  that  Army  Program  Managers  (PMs)  are  responsible  and  accountable  for 
the  life-cycle  management  of  their  assigned  programs.  Placing  responsibility  for  system  life-cycle  support  in  the 
hands  of  the  PM  dramatically  changed  their  relationship  with  MSCs.  PMs  became  MSC  customers  rather  than 
transition  organizations.  MSCs  had  to  rely  on  the  PMs  for  funding. 

In  August  2004,  Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technology  (ASA(ALT)) 
Claude  Bolton  signed  a  Memorandum  of  Agreement  (MOA)  (Bolton/Kern,  2004)  with  the  Commander  of 
AMC  (GEN  Paul  Kern)  creating  Life  Cycle  Management  Commands  (LCMCs)  of  the  AMC  MSCs  (Aviation 
and  Missile  Command  [AMCOM],  Tank  and  Automotive  Command  [TACOM],  Communications-Electronics 
Command  [CECOM],  Simulation  and  Training  Technology  Command  [STRICOM]).  The  MOA  encouraged 
synergy  by  aligning  MSCs  to  affiliated  Program  Executive  Offices  (PEOs)  and  PMOs. 

Did  these  policy  changes  improve,  diminish,  or  leave  unchanged  the  cost,  timeliness,  quality,  and  thoroughness 
of  weapon  system  software  sustainment  for  C4ISR  systems? 

Background 

In  the  past  60  years  of  modem  weapon  system  development,  the  U.S.  Congress,  Department  of  Defense 
(DoD),  and  the  military  services  have  modified  acquisition  policy  and  guidance  dozens  of  times.  The  cumulative 
impact  of  these  changes  on  weapon  system  development  and  acquisition  has  been  profound  and  lasting. 
Command,  Control,  Communications,  Computer,  Intelligence,  Surveillance,  and  Reconnaissance  (C4ISR) 
systems  used  by  today’s  warfighters  are  highly  software  intensive.  The  quality,  functionality,  and  performance  of 
the  software  within  these  systems  throughout  the  life  of  the  programs  must  meet  the  highest  possible  standards  to 
support  soldiers  in  garrison,  training,  exercise,  and  contingency  operations.  These  high  standards  must  be  achieved 
while  minimizing  costs,  a  difficult  challenge  as  the  United  States  considers  reduced  budgets  and  diminished 
operations  in  Iraq  and  Afghanistan. 

The  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  (USD[AT&L])  articulated 
requirements  for  “Obtaining  Greater  Efficiency  and  Productivity  in  Defense  Spending”  in  two  recent  memorandums 
(Carter,  June  28,  2010,  and  Sept.  14,  2010.)  Estimates  indicate  that  70  percent  of  life-cycle  costs  are  operations 
and  support,  while  only  30  percent  are  development  and  fielding  costs.  (Carter,  Sept.  14,  2010).  While  hardware 
costs  have  decreased  dramatically  to  less  than  20  percent  of  total  life-cycle  system  expenditures,  the  remaining 
80  percent  of  life-cycle  costs  comprise  all  non-hardware  expenses  such  as  training,  administrative  support, 
management,  and  software  (van  Vliet,  2002),  with  software  making  up  the  majority  of  these  remaining  costs.  A 
number  of  recognized  authorities  on  software  engineering  in  the  1980  timeframe  performed  studies  estimating 
sustainment  costs  for  commercial  software  systems  between  40  percent  and  75  percent  of  total  life-cycle  costs 
(Lientz  and  Swanson,  1980)  (Boehm,  1981).  To  the  dismay  of  these  same  experts,  these  statistics  have  not 
changed  much  in  the  years  since.  Bennetti  (1997)  indicates  that  later  surveys  revealed  software  sustainment 
comprises  between  40  percent  and  90  percent  of  total  life-cycle  software  expenditures. 

These  figures  indicate  that  software  sustainment,  and  the  cost  of  this  sustainment,  can  have  a  profound  impact 
on  the  success  or  failure  of  a  system  over  its  total  life.  Yet,  we  know  very  little  about  the  direct  impact  DoD  and 
military  services’  acquisition  policy  changes  have  on  this  critical  area  of  weapon  system  life-cycle  management. 
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Purpose  of  the  Study 

The  objective  of  this  study  was  to  collect  the  opinions  of  development,  sustainment,  and  user  organizations 
regarding  the  quality,  thoroughness,  timeliness,  and  cost-effectiveness  of  software  sustainment  support  for  the 
Army  C4ISR  enterprise  before  and  after  two  significant  U.S. 

Army  acquisition  policy  decisions  were  implemented  in  1997  and  2004.  Once  collected,  this  information  was 
analyzed  to  determine  significant  findings  regarding  the  perception  of  software  sustainment  before  and  after 
these  two  policy  decisions.  The  information  also  provided  perceptions  regarding  what  areas  of  software  support 
services  could  be  improved. 

A.  Research  Questions 

In  an  effort  to  address  these  issues  for  the  U.S.  Army  C4ISR  system  enterprise,  this  paper  seeks  to 
answer  the  following  specific  questions: 

(1)  Is  the  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among  users/units, 
PEOs/PMs  and  software  support  service  providers  better  or  worse  since  the  1997  ASA(RDA)  decision? 

(2)  Is  the  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among  users/units, 
PEOs/PMs,  and  software  support  service  providers  better  or  worse  since  the  2004  MOA  between  the 
ASA(ALT)  and  CG  AMC? 

(3)  What  is  the  current  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among 
users/units,  PEOs/PMs,  and  software  support  service  providers? 

B.  Research  Hypotheses 

HI:  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of 
software  sustainment  support  for  C4ISR  systems  after  the  1997  ASA(RDA)  decision  compared  to  the 
perception  of  C4ISR  software  sustainment  support  prior  to  1997. 

H2:  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of 
software  sustainment  support  for  C4ISR  systems  after  the  2004  ASA(ALT)/AMC  MOA  compared  to 
the  perception  of  C4ISR  software  sustainment  support  prior  to  2004. 

H3:  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of 
software  sustainment  support  for  C4ISR  systems  today. 

Significance  of  the  Study 

This  study  contributes  information  about  the  impact  DoD  and  military  services’  acquisition  policy  changes 
have  on  life-cycle  software  support  and  thus,  overall  system  life-  cycle  cost  and  effectiveness.  The  study  is  based 
on  the  perception  of  quality,  timeliness,  cost,  and  thoroughness  of  software  sustainment  support  by  Army  C4ISR 
PM,  user/operator,  and  software  support  personnel. 

Overview  Methodology 

The  study  was  designed  to  survey  personnel  involved  in  the  business  of  developing,  acquiring,  supporting,  and 
operating  software  intensive  Army  C4ISR  systems  and  to  solicit  their  opinions  on  the  quality,  timeliness,  cost, 
and  thoroughness  of  software  sustainment  support.  Survey  participants  were  divided  into  three  groups:  those 
involved  with  Army  C4ISR  systems  prior  to  the  1998  decision  to  make  PMs  responsible  for  system  life-cycle 
management,  those  involved  with  Army  C4ISR  systems  prior  to  the  2004  decision  to  create  AMC  Life  Cycle 
Management  Commands  (LCMCs),  and  those  involved  with  Army  C4ISR  systems  since  the  2004  decision. 

The  survey  vehicle  was  posted  on  an  Army  Knowledge  Online  (AKO)  Web  page  with  unlimited  access. 
Emails  were  sent  to  members  of  the  Army  C4ISR  community,  asking  participants  to  complete  the  online  survey. 
Survey  results  were  analyzed  to  quantify  the  perception  participants  have  of  software  sustainment  support  during 
these  three  periods.  Participants  were  also  asked  to  select  areas  in  which  current  software  sustainment  support 
should  improve. 
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Scope,  Limitations,  and  Delimitations 

The  scope  of  the  study  was  constrained  in  several  areas.  This  research  paper  was  limited  to  U.S.  Army  C4ISR 
systems  supported  by  Team  C4ISR  comprised  of  PEO  C3T  and  its  Program  Management  Offices  (PMO);  PEO 
IEW&S  and  its  PMOs;  PEO  EIS  and  its  PMOs;  and  the  CECOM  LCMC  Software  Engineering  Center  (SEC). 
The  study  was  intended  to  include  U.S.  Army  C4ISR  system  user  units,  staff,  and  operators.  Similar  systems  in 
other  military  services  or  DoD  agencies  are  outside  the  scope  of  this  research  paper.  A  precise  estimate  of  the 
size  of  the  Army  C4ISR  community  was  not  available  and  a  random  process  for  sampling  the  community  was 
not  possible.  This,  as  well  as  the  limited  response  to  the  survey,  prevented  collection  of  a  statistically  significant 
number  of  survey  responses. 

This  research  paper  does  not  address  objective  measures  of  system  software  functionality,  performance,  quality, 
and  cost  of  software  support  over  the  program  life.  The  study  is  focused  on  subjective  opinion  or  perception  of 
these  criteria  by  PM,  user,  and  software  support  staff.  While  the  study  may  answer  research  questions  about  the 
perception  of  software  sustainment  support  for  Army  C4ISR  systems  before  and  after  significant  policy  decisions, 
the  study  does  not  isolate  these  decisions  as  the  only  reasons  for  differences  in  the  perception  of  software  support 
during  these  periods.  Other  factors  may  have  affected  perceptions.  For  example,  in  2001,  the  United  States  began 
Operation  Enduring  Freedom,  launching  an  attack  on  terrorist  forces  in  Afghanistan  in  response  to  9/1 1  attacks  in 
the  United  States.  In  2003,  the  United  States  began  Operation  Iraqi  Freedom.  These  two  conflicts  continue  today. 
The  nature  and  provision  of  software  sustainment  support  for  Army  C4ISR  systems  during  these  two  conflicts 
may  have  influenced  perceptions  of  software  sustainment  support  compared  to  perceptions  prior  to  1998. 

Assumptions 

Providing  data  regarding  the  quality,  timeliness  and  cost  of  software  support  is  not  all  that  is  required  to 
address  software  sustainment  support.  Perceptions  of  PM,  user,  and  software  support  personnel  regarding  the 
quality,  timeliness,  cost,  and  thoroughness  of  software  sustainment  support  are  as  important  to  an  assessment 
of  these  services  as  are  quantitative  measures  of  these  factors.  A  useful  analysis  of  the  perception  of  software 
sustainment  support  within  the  Army  C4ISR  community  is  possible  without  a  statistically  significant  number  of 
survey  respondents.  While  not  meeting  the  statistical  criteria  for  significance,  data  collected  on  perceptions  of 
software  support  are  useful  in  arriving  at  conclusions  for  this  study. 

While  not  the  only  factors  affecting  perceptions  of  software  sustainment  support  for  Army  C4ISR  systems, 
the  two  policy  decisions  in  question  significantly  influenced  support.  Therefore,  the  inability  to  isolate  key  policy 
decisions  as  the  only  factors  causing  differences  in  the  perception  of  software  sustainment  support  for  Army 
C4ISR  systems  does  not  diminish  the  value  of  study  results. 

Definition  of  Key  Terms 

See  Glossary  of  Acronyms  and  Terms. 

Organization  of  the  Research  Paper 

Chapter  1  introduced  the  background,  problem  statement  and  purpose  of  the  study  with  research  questions  and 
hypotheses.  The  introduction  also  provides  a  description  of  the  significance  of  the  study,  an  overview  of  the  study 
methodology,  and  scope,  limitations,  and  delimitations  of  the  study.  Chapter  2  is  a  literature  review  describing 
professional  and  academic  information  sources  and  DoD  and  military  service  policies  regarding  software 
development  and  software  life-cycle  management.  Chapter  3  describes  the  research  methodology  including  the 
research  perspective  and  design,  survey  participants,  the  survey  instrument,  data  collection  and  analysis.  Chapter 
4  describes  survey  results  and  analyzes  them  in  terms  of  the  study  hypotheses.  Chapter  5  provides  a  summary  and 
conclusions,  suggesting  implications  for  further  research. 
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CHAPTER  2 

LITERATURE  REVIEW 


Introduction 

This  chapter  summarizes  results  of  a  review  of  literature  applicable  to  the  research  questions.  The  literature 
review  focused  on  academic,  professional,  and  government  documentation  and  studies  regarding  software 
engineering,  software  support,  and  life-cycle  software  costs.  Original  studies  and  surveys  documenting  the  cost 
of  life-cycle  software  support  predominantly  were  from  the  early  1980s.  These  studies  continued  to  be  referenced 
by  other  academic  books  and  studies  well  into  the  late  1990s.  Separate  studies  addressing  costs  and  processes  for 
supporting  military  software-based  systems  relied  heavily  on  these  same  studies. 

Army  policies  implement  directives  and  memoranda  issued  by  military  leaders.  These  documents  validate 
changes  to  Army  acquisition  policies  which  serve  as  the  impetus  behind  the  three  research  questions.  References 
used  are  roughly  categorized  into  (1)  professional  and  academic  sources  in  the  area  of  software  engineering,  and 
(2)  DoD  and  Army  acquisition  policy  documents. 

Professional  and  Academic  Sources 

Academic  sources  address  studies  involving  support  to  commercial  software  information  technology  systems. 
These  studies  compare  the  cost  of  supporting  software  after  implementation  to  the  overall  cost  of  the  software 
program.  The  literature  search  involved  software  support  costs  for  commercial  systems  from  studies  in  the  early 
1980s,  such  as  Boehm  (1981)  and  Lientz  and  Swanson  (1980).  Existing  studies  and  data  regarding  sustainment  of 
military  C4ISR  systems  focus  on  the  cost  of  system  support  compared  to  life-cycle  costs  of  a  program.  Estimates 
of  C4ISR  system  software  sustainment  are  based  on  commercial  models  and  historical  data,  such  as  U.S.  Air 
Force  Software  Technology  Support  Center  Cost  Analysis  Group  (2002). 

Department  of  Defense  and  Military  Service  Directives  and  Policies 

Department  of  Defense  (DoD)  and  Army  acquisition  policy  documents  articulate  official  decisions  and 
guidelines  affecting  weapon  system  acquisition  and  the  software  embedded  within  these  systems.  Such  documents 
as  AR  70-1  (Dec.  15,  1997)  and  the  Carter  memo  (Sept.  14,  2010)  are  included.  Available  sources  did  not  address 
the  quality,  timeliness,  or  thoroughness  of  sustainment  support  for  either  commercial  or  military  systems. 
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CHAPTER  3 

RESEARCH  METHODOLOGY 


Introduction 

This  chapter  describes  the  methodology,  design,  survey  population,  research  instrument,  data  collection 
and  analysis  used  in  the  research.  The  study  makes  inquiries  about  the  affect  Army  acquisition  policy  changes 
have  on  software  sustainment  support.  Specifically,  do  these  policy  changes  and  organizational  alignments 
improve,  diminish,  or  leave  unchanged  the  cost,  timeliness,  quality,  and  thoroughness  of  weapon  system  software 
sustainment  for  C4ISR  systems? 

In  April  1997,  the  ASA(RDA),  Mr.  Decker,  issued  a  memorandum  directing  that  Army  PMs  are  responsible 
and  accountable  for  the  life-cycle  management  of  their  assigned  programs.  In  August  2004,  the  ASA(ALT), 
Mr.  Bolton,  and  Commander  of  AMC,  GEN  Kern,  signed  a  memorandum  of  agreement  creating  Life  Cycle 
Management  Commands  (LCMCs)  of  the  AMC  MSCs  in  order  to  align  AMC  MSCs  with  appropriate  Program 
Executive  Offices  (PEOs).  The  research  seeks  to  understand  the  perception  of  the  quality,  timeliness,  cost,  and 
thoroughness  of  software  sustainment  support  for  Army  C4ISR  systems  following  these  two  policy  decisions. 

Research  Perspective  and  Design 

The  research  study  was  a  phenomenological  approach,  aimed  at  understanding  the  perception  of  the  quality, 
timeliness,  cost,  and  thoroughness  of  software  sustainment  support  within  the  Army  C4ISR  development, 
operations,  and  support  community.  Understanding  the  perception  of  software  sustainment  support  before  and 
after  the  two  significant  acquisition  policy  decisions  were  made  adds  information  as  to  the  effect  such  policy 
decisions  have  on  life-cycle  software  support. 

The  design  selected  for  this  research  was  a  survey  of  personnel  engaged  in  either  the  development,  support, 
or  operation  of  Army  C4ISR  system  software.  The  uniform  resource  locator  (URL)  address  for  the  posted  survey 
form  was  provided  in  emails  to  leaders  within  the  Army  C4ISR  community  with  an  invitation  to  voluntarily  access 
the  Website  and  complete  the  survey,  as  well  as  forward  the  email  to  individuals  and  staff  they  believed  should 
complete  the  survey  form. 

Research  Questions  and  Hypotheses 

A.  Research  Questions 

(1)  Is  the  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among  users/units, 
PEOs/PMs  and  software  support  service  providers  better  or  worse  since  the  1997  ASA(RDA)  decision? 

(2)  Is  the  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among  users/units, 
PEOs/PMs,  and  software  support  service  providers  better  or  worse  since  the  2004  MOA  between  the 
ASA(ALT)  and  CG  AMC? 

(3)  What  is  the  current  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among 
users/units,  PEOs/PMs,  and  software  support  service  providers? 

B.  Hypotheses 

(HI)  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of 
software  sustainment  support  for  C4ISR  systems  after  the  1997  ASA(RDA)  decision  compared  to  the 
perception  of  C4ISR  software  sustainment  support  prior  to  1997. 

(H2)  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of 
software  sustainment  support  for  C4ISR  systems  after  the  2004  ASA(ALT)/AMC  MOA  compared  to 
the  perception  of  C4ISR  software  sustainment  support  prior  to  2004. 

(H3)  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of 
software  sustainment  support  for  C4ISR  systems  today. 
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Participants  and  Population 

The  survey  population  consisted  of  military,  government  civilian,  and  contractor  personnel  engaged  in  the 
development,  support,  and/or  operation  of  Army  C4ISR  systems.  This  group  included  leadership,  program 
management,  and  staff  from  PEO  C3T,  PEO  IEW&S,  and  PEO  EIS.  Software  sustainment  support  leaders  and 
staff  from  the  Army  CECOM  LCMC  Software  Engineering  Center  (SEC)  were  included,  as  well  as  operators  of 
C4ISR  systems  within  Army  tactical  units  around  the  globe. 

Unit  of  Analysis  and  Research  Variables 

The  individual  survey  participant  was  the  unit  of  analysis.  There  were  three  primary  variables  used  in  the 
analysis:  the  time  period  the  respondent  began  involvement  with  Army  C4ISR  systems,  the  current  role  of  the 
respondent,  and  respondent  scoring  of  software  sustainment  support.  The  time  period  variable  had  one  of  three 
possible  values.  The  earliest  was  the  period  prior  to  the  ASA(ALT)  decision  to  make  PMs  responsible  for  system 
life-cycle  management,  or  prior  to  1998.  The  next  time  period  was  after  the  ASA(RDA)  decision  but  prior  to  the 
MOA  between  the  ASA(ALT)  and  Commander  AMC,  or  between  1998  and  2004.  The  third  time  period  was  any 
time  after  2004.  There  were  five  possible  roles  for  participants.  One  was  government  and  military  members  of  a 
C4ISR  PEO  or  PMO.  Another  was  government  and  military  members  of  a  software  sustainment  support  activity. 
In  this  case,  CECOM  LCMC  SEC.  Government  and  military  operators  of  Army  C4ISR  systems  were  a  category. 
The  survey  also  permitted  a  category  of  Other.  Survey  participants  scored  software  sustainment  support  based 
on  qualitative  values  listed  as  Excellent,  Good,  Adequate,  Poor,  and  Very  Poor.  These  qualitative  scores  were 
translated  into  numerical  values  of  5,  4,  3,  2,  and  1  for  purposes  of  calculating  statistical  averages. 

Research  Instrument 

A  survey  was  developed  for  use  in  collecting  information  on  the  variables.  The  survey  form  was  posted  on  the 
Army  Knowledge  On-Line  (AKO)  Website  accessible  by  personnel  who  were  given  the  uniform  resource  locator 
(URL)  address.  The  survey  announcement  was  emailed  to  appropriate  PEO,  PM,  CECOM  SEC,  and  user  units 
and  staff  requesting  information  on  opinions  about  the  quality,  timeliness,  thoroughness,  and  cost-effectiveness 
of  software  sustainment  support  for  C4ISR  systems.  Emails  also  asked  recipients  to  forward  the  request  for 
participation  to  other  staff  and  members  of  the  Army  C4ISR  community.  Survey  results  were  posted  to  the  survey 
Website.  Survey  questions  and  response  options  are  at  Appendix  A. 

Pilot  Study 

A  pilot  survey  form  was  provided  to  five  individuals  to  ensure  survey  questions  and  response  options  were 
sufficient  to  answer  research  questions.  The  pilot  resulted  in  modifying  the  survey  form  to  permit  inclusion  of 
C4ISR  contract  personnel  and  add  more  areas  of  possible  software  sustainment  support  improvement  selections 
in  Question  6.  The  pilot  group  also  advised  against  posting  the  survey  on  a  commercial  survey  Website.  They 
indicated  government  and  military  personnel  would  be  hesitant  to  respond  to  such  a  survey  on  a  commercial 
Website.  As  a  result,  the  survey  was  posted  on  an  Army  Knowledge  On-Line  (AKO)  Web  page.  The  survey  form 
was  further  modified  to  accommodate  restrictions  mandated  by  the  AKO  survey  development  tool. 

Data  Collection  Procedures 

Survey  results  were  collected  on-line  at  the  AKO  Website  where  the  survey  instrument  was  posted.  AKO 
saved  and  listed  answers  for  each  of  the  seven  survey  questions  for  each  respondent.  Results  posted  on  the  AKO 
web  site  were  retrieved  and  documented  in  an  off-line  spreadsheet.  The  spreadsheet  tools  were  used  to  sort  and 
count  responses  in  order  to  summarize  and  statistically  analyze  results. 
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Data  Collection  and  Statistical  Analysis 

Spread  sheets  used  to  capture  survey  data  were  also  used  to  sort  and  summarize  data  describing  key  variables. 
Summarized  information  includes  such  figures  as  total  number  of  respondents;  number  and  percentage  of 
respondents  with  C4ISR  experience  prior  to  1998,  between  1998  and  2004,  since  2004;  number  and  percentage 
of  respondents  who  are  PEO/PM,  software  support,  and  user  personnel.  Information  used  to  describe  the  survey 
population  partitions  set  the  stage  for  analyzing  data  collected  about  the  quality,  timeliness,  cost,  and  thoroughness 
of  software  support.  Scores  of  software  support  by  survey  participants  were  totaled,  graphed,  and  numerical 
averages  calculated.  Similar  total  and  statistical  calculations  were  generated  for  software  support  scores  for  each 
survey  population  partition  to  assess  the  perception  of  software  support  according  to  type  C4ISR  community 
members. 

Setting  and  Environment 

Email  requests  to  log  into  and  participate  in  the  AKO  survey  were  sent  to  key  personnel  within  the  Army 
C4ISR  PEO/PM  and  CECOM  SEC  activities.  Emails  were  sent  to  participants’  AKO  addresses.  Participants  likely 
completed  the  surveys  from  their  office  environment  on  government  installations.  These  personnel  were  also 
asked  to  forward  the  email  to  their  staff  and  customers  with  a  request  to  participate  in  the  survey.  This  method  of 
proliferating  the  request  for  Army  C4ISR  community  participants  was  intended  to  increase  the  number  of  survey 
respondents.  Unfortunately,  the  number  of  respondents  was  too  low  to  result  in  a  statistically  significant  sample 
of  the  Army  C4ISR  community.  Results,  while  not  statistically  significant,  provide  useful  information  toward 
answering  the  research  questions. 

Bias  and  Error 

Lack  of  random  sampling,  the  low  number  of  respondents,  and  disproportionate  number  of  responses  from 
some  sectors  of  Army  C4ISR  community  indicate  overall  scoring  statistics  may  be  biased.  Results  and  conclusions 
address  these  possible  biases  in  the  data. 

Validity 

Questions  in  the  research  instrument  aimed  at  partitioning  the  survey  population  according  to  their  experience 
during  timeframes  affected  by  the  study  policy  decisions  assures  that  later  questions  regarding  quality,  timeliness, 
cost,  and  thoroughness  of  software  sustainment  support  represent  the  content  that  the  research  is  intended  to 
address. 

Summary 

As  a  study  of  the  adequacy  of  Army  C4ISR  software  sustainment  support  after  implementing  significant 
acquisition  policy  decisions,  the  research  is  well  situated  to  use  a  phenomenological  approach,  collecting 
information  through  a  population  survey.  Research  questions,  and  resulting  hypotheses,  focus  the  study  on 
specific  issues.  The  survey  instrument  used  a  relatively  straight-forward  seven-question  format  to  establish  the 
time-frame  during  which  population  members  had  been  involved  in  C4ISR  support  and  to  which  population 
subgroup  they  belonged  .  The  survey  questions  were  developed  to  directly  address  variables  needed  to  answer 
research  questions.  The  survey  was  posted  on  a  Website  easily  accessible  by  potential  survey  respondents.  Data 
collected  was  sorted,  summarized,  and  analyzed  using  common  statistical  analysis  tools. 

Results  will  show  a  weakness  not  in  the  methodology,  but  in  the  execution  of  the  approach  to  collecting  data. 
The  entire  population  of  possible  respondents  could  not  be  notified  about  the  survey  and  a  random  sample  set 
of  survey  participants  was  not  possible.  Instead,  key  personnel  within  the  population  were  notified  and  asked  to 
proliferate  the  survey  to  other  potential  members.  As  a  volunteer,  rather  than  a  mandated,  activity,  completion  and 
further  distribution  of  the  survey  was  limited.  Survey  respondents  were  too  few  and  lacked  necessary  randomness 
to  achieve  what  could  be  considered  a  statistically  significant  portion  of  the  population.  A  disproportionate 
number  of  respondents  are  from  certain  sectors  of  the  population.  While  these  limitations  reduce  the  value  of  the 
study  in  reflecting  the  perspectives  of  the  entire  population,  they  do  not  diminish  the  value  of  assessing  data  about 
the  restricted  population  sectors  for  which  results  can  be  analyzed. 
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CHAPTER  4 

DATA  ANALYSIS  AND  RESULTS 


Introduction 

This  chapter  presents  results  of  the  study.  Results  are  presented  in  four  sections.  The  first  section  describes  the 
survey  respondents  based  on  the  number  of  participants  according  to  the  timeframe  they  joined  the  Army  C4ISR 
community  and  the  type  of  C4ISR  activities  in  which  they  are  currently  engaged  (i.e.  user/operator,  PEO/PM, 
software  sustainment  support.)  The  second  section  provides  scoring  of  software  sustainment  support  based  on 
survey  participant  time  frame.  The  third  section  provides  scoring  of  software  sustainment  support  based  on  type 
of  C4ISR  respondent.  The  fourth  section  provides  overall  scoring  results. 

A  spreadsheet  recording  survey  results  is  at  Appendix  B.  To  calculate  mean  average  scores,  numerical  values 
were  assigned  to  each  score.  A  score  of  excellent  was  weighted  as  5,  good  as  4,  adequate  as  3,  poor  as  2,  and  very 
poor  as  1 . 

Participant  Profile 

A  total  of  42  survey  forms  were  completed  for  this  study.  Participants  were  asked  to  identify  which  role  they 
fulfill  in  the  Army  C4ISR  community.  Results  are  shown  in  Table  1  and  Figure  1,  below.  A  disproportionately 
large  number  of  participants  are  designated  as  Software  Sustainment  Support  Activity  and  PEO/PM  personnel, 
while  a  disproportionately  small  number  of  respondents  are  User/Operator  and  Contractor  personnel.  Because 
PEO/PM,  Software  Sustainment  Support  Activity,  and  Other  categories  of  participants  make  up  93  percent  of  the 
survey  results,  analysis  by  type  role  of  respondents  focuses  on  these  three  categories. 

Participants  were  also  asked  to  identify  when  they  became  a  part  of  the  Army  C4ISR  community.  This  data 
assists  in  isolating  those  respondents  who  are  able  to  provide  their 


USER/OPERATOR 

1 

PEO/PM 

10 

SSSA 

23 

CONTRACTOR 

2 

OTHER 

6 

TOTAL 

42 

Table  1,  Participants  by  Role 


Participants  by  Role 

2% 


■  USER/OPERATOR 

■  PEO/PM 

■  SSSA 

■  CONTRACTOR 
OTHER 


Figure  1,  Participants  by  Role 
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perceptions  of  software  support  prior  to  the  1998  ASA(RDL)  direction  that  PMs  are  responsible  for  life-cycle 
management  of  systems,  those  able  to  provide  their  perceptions  of  software  support  between  1998  and  2004 
after  the  ASA(RDL)  direction,  and  those  after  2004  when  the  Army  ASA(ALT)  and  CG  AMC  created  LCMCs 
The  portion  of  survey  respondents  who  were  part  of  the  Army  C4ISR  community  prior  to  1998  is  larger  than 
anticipated,  while  the  portion  of  respondents  in  the  other  two  categories  is  smaller  than  anticipated. 


BEFORE  1998 

28 

1998  -  2004 

5 

AFTER  2004 

TOTAL 

9 

42 

Table  2,  Participants  by  Year 

Participants  by  Year 


■  BEFORE  1998 

■  1998-2004 

■  AFTER  2004 


Figure  2,  Participants  by  Year 


Ratings  by  Timeframes 

Participants  who  were  part  of  the  Army  C4ISR  community  prior  to  the  1998  ASA(RDA)  decision-  making 
PMs  responsible  for  system  life-cycle  management  scored  software  sustainment  support  as  shown  in  Table  3  and 
Figure  3,  below.  The  weighted  mean  score  for  software  sustainment  support  prior  to  1998  was  3.46,  or  between 
adequate  and  good.  Figure  3  provides  a  histogram  of  scoring  frequency,  illustrating  that  the  number  of  participants 
scoring  software  support  as  adequate  and  good  were  equal  with  a  significantly  smaller  number  of  excellent,  poor, 
and  very  poor  scores. 


FREQ 

WTD 

EXCELLENT 

3 

15 

GOOD 

11 

44 

ADEQUATE 

11 

33 

POOR 

2 

4 

VERY  POOR 

1 

1 

TOTAL 

WEIGHTED 

MEAN 

28 

3.46 

Table  3,  Scores  Prior  to  1998  (All  Roles) 
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Frequency  of  Scores  Prior  to  1998 
(All  Roles) 


EXCELLENT  GOOD  ADEQUATE  POOR  VERY  POOR 


Figure  3,  Frequency  of  Scores  Prior  to  1998  (All  Roles) 


Participants  who  were  part  of  the  Army  C4ISR  community  between  1998  and  2004  scored  software  sustainment 
support  as  shown  in  Table  4  and  Figure  4,  below.  The  weighted  mean  score  for  software  sustainment  support 
between  1998  and  2004  was  3.74,  again  between  adequate  and  good,  similar  to  pre-1998  scores.  However,  Figure 
4  provides  a  notably  different  histogram  of  scoring  frequency,  illustrating  that  the  number  of  participants  scoring 
software  support  as  good  predominated  scoring  with  substantially  fewer  scores  at  higher  and  lower  levels,  creating 
a  more  normal  distribution  of  scores. 


FREQ 

WTD 

EXCELLENT 

2 

10 

GOOD 

20 

80 

ADEQUATE 

8 

24 

POOR 

1 

2 

VERY  POOR 

0 

0 

TOTAL 

WEIGHTED 

MEAN 

31 

3.74 

Table  4,  Scores  Between  1998  and  2004  (All  Roles) 
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Frequency  of  Scores  Between  1998 
and  2004  (All  Roles) 


■  - 

EXCELLENT  GOOD  ADEQUATE  POOR  VERY  POOR 


Figure  4,  Frequency  of  Scores  Between  1998  and  2004  (All  Roles) 


Participants  who  were  part  of  the  Army  C4ISR  community  after  the  2004  MOA  between  ASA(ALT)  and  CG 
AMC  to  create  Life  Cycle  Management  Commands,  scored  software  sustainment  support  as  shown  in  Table  5  and 
Figure  5,  below.  The  weighted  mean  score  for  software  sustainment  support  prior  to  1998  was  3.9,  slightly  more 
than  the  average  score  between  1998  and  2004.  Figure  5  shows  a  histogram  of  scoring  frequency  very  much  like 
the  scoring  distribution  between  1998  and  2004. 


FREQ 

WTD 

EXCELLENT 

6 

30 

GOOD 

27 

108 

ADEQUATE 

8 

24 

POOR 

1 

2 

VERY  POOR 

0 

0 

TOTAL 

WEIGHTED 

MEAN 

42 

3.90 

Table  5,  Scores  After  2004  (All  Roles) 
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Frequency  of  Scores  After  2004 
(All  Roles) 

25 

20 

15 

10 

c 

3 

n 

EXCELLENT 

GOOD 

ADEQUATE  POOR  VERY  POOR 

Figure  5,  Frequency  of  Scores  After  2004  (All  Roles) 


Ratings  by  Participant  Type 

Of  the  total  participant  surveys  completed,  24  percent  were  from  PEO/PM  staff,  55  percent  were  from 
Software  Sustainment  Support  Activities  (SSSA),  and  14  percent  were  from  participants  labeling  themselves  as 
Other.  Together,  these  three  sectors  of  the  Army  C4ISR  population  sample  constitute  93  percent  of  the  survey 
respondents.  Table  6  and  Figure  6  illustrate  the  weighted  average  scoring  and  distribution  of  scores  for  software 
support  in  the  three  time  frames  for  these  three  sectors  of  the  Army  C4ISR  community. 

Prior  to  1998,  average  scoring  for  all  respondents  was  3.46.  During  this  same  time  period,  average  scoring  by 
PEO/PM  participants  was  3.5;  for  SSSA  participants  it  was  3.53,  and  for  Other  participants  it  was  3.33.  Except 
for  outlier  scores  above  and  below  good  and  adequate,  the  frequency  of  PEO/PM  scoring  during  this  time  frame 
was  similar  to  distributions  within  the  overall  participant  sample.  Like  PEO/PM  participants,  the  frequency 
of  scores  for  SSSA  participants  during  this  time  was  similar  to  overall  scoring,  but  more  widely  distributed. 
Distribution  of  scores  for  Other  participants  during  this  time  was  markedly  different  than  other  sectors,  showing 
a  stronger  tendency  to  score  software  support  as  adequate,  rather  than  good. 

Between  1998  and  2004,  average  scoring  for  all  respondents  was  3.74.  During  this  time  period,  average 
scoring  by  PEO/PM  respondents  was  4;  for  SSSA  respondents  it  was  3.76,  and  for  Other  respondents  the  average 
was  3.25.  Because  all  PEO/PM  respondents  scored  software  support  between  1998  and  2004  as  good,  scoring 
distribution  was  limited  to  a  single  rating.  As  in  scoring  prior  to  1998,  SSSA  respondents  had  a  distribution  of 
scores  similar  to  the  overall  population  sample,  but  more  widely  distributed  than  scoring  by  PEO/PM  respondents. 
Because  Other  respondents  scored  software  support  between  1998  and  2004  similar  to  scoring  prior  to  1998, 
score  distribution  tended  toward  adequate,  rather  than  good. 

After  2004,  average  scoring  for  all  respondents  was  3.9,  slightly  improved  over  previous  periods.  Average 
scoring  for  PEO/PM  participants  during  this  time  was  3.7 ;  for  SSSA  participants  it  was  3.96;  for  Other  participants 
average  scoring  in  this  time  frame  was  3.75.  PEO/PM  participant  scoring  distribution  after  2004  shows  a  slightly 
greater  tendency  to  score  software  support  in  the  lower  ratings  than  the  overall  population  sample.  The  distribution 
of  SSSA  scores  after  2004  was  similar  to  that  of  overall  scoring,  but  had  a  noticeably  greater  number  of  higher 
scores  than  did  the  PEO/PM  sector.  The  distribution  of  Other  participant  scoring  showed  a  marked  change  after 
2004,  with  average  scoring  shifting  from  3.25  in  the  previous  time-frame  to  3.83.  Although  different  than  other 
profiles,  the  distribution  of  Other  scores  after  2004  is  much  closer  to  overall  scores  and  PEO/PM  and  SSSA 
sectors  than  in  previous  periods. 

In  the  overall  survey  population,  average  scores  tended  to  increase  for  each  of  the  three  time  periods  under 
study.  This  trend  was  influenced  by  each  of  the  three  sectors  that  make  up  93  percent  of  survey  responses  from 
the  Army  C4ISR  community.  One  exception  was  the  PEO/PM  sector  with  an  average  rating  between  1998  and 
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2004  of  4,  while  average  scoring  after  2004  was  3.7.  Another  exception  was  the  Other  sector  of  participants, 
with  average  scores  of  3.33  prior  to  1998,  3.25  between  1998  and  2004,  and  3.83  after  2004.  Score  distributions 
for  these  three  sectors  varied.  Score  distributions  for  the  SSSA  sector  tended  to  be  more  widely  distributed  than 
either  the  PEO/PM  or  Other  sectors,  indicating  the  SSSA  population  has  a  more  diverse  perspective  of  software 
support,  while  the  other  two  sectors  have  a  more  homogenous  perspective  of  software  support. 


PRIOR  TO  1998 

BETWEEN  1998  AND  2004 

AFTER  2004 

PEO/PM 

SSSA 

OTHER 

PEO/PM 

SSSA 

OTHER 

PEO/PM 

SSSA 

OTHER 

EXCELLENT 

0 

3 

0 

0 

2 

0 

0 

5 

0 

GOOD 

3 

7 

1 

8 

10 

1 

8 

12 

5 

ADEQUATE 

3 

4 

2 

0 

4 

3 

1 

6 

1 

POOR 

0 

2 

0 

0 

1 

0 

1 

0 

0 

VERY  POOR 

WEIGHTED 

0 

1 

0 

0 

0 

0 

0 

0 

0 

MEAN 

3.50 

3.53 

3.33 

4.00 

3.76 

3.25 

3.70 

3.96 

3.83 

Table  6,  Scores  for  Each  Time  Period  (PEO/PM,  SSSA,  OTHER) 
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Figure  6,  Frequency  of  Scores  for  Each  Time  Period  (PEO/PM,  SSA,  Other) 

Suggested  Software  Support  Improvements 

Respondents  were  asked  to  select  areas  in  which  they  believed  software  sustainment  support  should  improve. 
Frequency  of  suggestions  are  graphed  in  the  Pareto  Chart  at  Figure  7,  below.  Over  all  respondents,  improving 
the  cost  of  field  support  was  the  most  frequent  suggestion  by  a  substantial  margin.  The  second  most  frequent 
suggestion  was  improving  timeliness  of  system  accreditation  with  improving  the  timeliness  of  software  upgrades  a 
close  third.  Two  categories  tied  for  fourth:  improving  the  quality  and  thoroughness  of  field  support  and  improving 
the  quality  of  software  upgrades.  Tied  for  fifth-highest  frequency  were  improving  the  quality  and  availability  of 
online  and  help  desk  support. 
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Figure  7,  Suggested  Software  Support  Improvements 

Figure  8  graphs  the  frequency  of  suggestions  according  to  the  three  largest  participant 
sectors  that  make  up  93  percent  of  the  respondents.  PEO/PM  respondents  suggested  improving  the  timeliness  of 
accreditation  with  the  most  frequency,  followed  by  the  timeliness  of  software  upgrades  and  cost  of  software  field 
support  suggested  second  and  third  most  frequently.  Two  areas  of  improvement  were  suggested  with  the  fourth- 
highest  frequency:  quality  of  software  upgrades,  and  cost  of  system  accreditation.  The  fifth-highest  frequency 
suggestion  was  to  improve  the  quality  of  on  line  support. 

SSSA  participants  selected  improving  the  cost  of  field  support  most  frequently  by  a  large  margin.  The  second- 
highest  frequency  SSSA  suggestions  were  improving  the  quality  and  thoroughness  of  field  support  and  quality 
and  availability  of  help  desk  support.  Two  improvement  areas  were  suggested  by  SSSA  personnel  the  third  most 
frequently:  quality  and  availability  of  online  support  and  timeliness  of  system  accreditation.  The  fourth  most 
frequently  suggested  area  of  improvement  was  the  timeliness  of  software  upgrades.  SSSA  respondents  called 
out  six  areas  of  improvement  the  fifth  most  frequently:  quality  of  software  upgrades,  timeliness  of  field  support, 
cost  of  help  desk  support,  quality  of  information  assurance,  cost  of  information  assurance,  and  cost  of  system 
accreditation. 

Other  participants  called  out  three  areas  of  improvement  most  frequently:  quality  of  software  upgrades, 
timeliness  of  software  upgrades,  and  timeliness  of  system  accreditation.  Improving  the  cost  of  help  desk  and  quality 
of  system  accreditation  were  called  out  the  second  most  frequently  by  Other  participants.  Seven  improvement 
areas  were  called  out  the  third  most  frequently:  cost  of  software  upgrades,  cost  of  field  support,  timeliness  of  field 
support,  quality  and  availability  of  online  support,  quality  and  availability  of  help  desk  support,  implementation 
of  information  assurance,  and  cost  of  system  accreditation. 

Participant  Comments 

Participants  were  asked  to  provide  comments  on  Army  C4ISR  software  sustainment  support.  Comments 
varied.  However,  there  were  four  comments  each  that  focused  on  concerns  about  the  cost  of  software  upgrades 
and  the  capabilities  of  field  engineers  and  technicians.  Two  comments  each  addressed  concerns  about  scheduling 
or  timeliness  of  support  and  providing  software  that  permits  soldiers  to  load  and  install  upgrades  and  security 
patches.  A  complete  listing  of  all  comments  is  at  Appendix  C. 
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Figure  9,  Suggested  Software  Support  Improvements  (PEO/PM,  SSSA,  Other) 


Summary  of  Results 

Forty-two  individuals  from  the  Army  C4ISR  community  responded  to  the  survey  request.  Fifty-five  percent 
of  the  respondents  were  from  the  SSSA,  24  percent  from  PEO/PM  staff,  14  percent  labeled  themselves  as  Other. 
The  remaining  7  percent  was  split  between  User/Operator  and  Contractor  personnel.  Software  support  scoring  by 
all  participants  prior  to  the  1998  decision  by  the  ASA(RDA)  had  a  numerical  average  of  3.46  between  good  and 
adequate.  Averages  for  the  two  subsequent  time  periods  between  1998  and  2004  and  after  the  2004  ASA(ALT) 
and  CG  AMC  MOA  increased  to  3.74  and  3.9,  respectively.  While  these  averages  indicate  a  gradual  increase 
in  scoring  over  the  three  time  periods,  the  differences  are  not  substantial.  Distribution  of  overall  scoring  was 
somewhat  dual-mode  prior  to  1998  with  respondents  equally  split  between  good  and  adequate.  Score  frequencies 
were  more  normally  distributed  in  the  two  subsequent  time  periods. 

As  the  largest  participating  sector  of  the  Army  C4ISR  community,  SSSA  drove  scoring  that  tracked  overall 
scoring  distributions  in  two  of  the  three  time  periods.  Prior  to  1998,  the  dual-mode  scoring  distribution  differed 
from  SSSA’s  normal  distribution  because  of  the  large  number  of  PEO/PM  and  Other  participants  scoring  in  the 
lower  rating  of  adequate.  In  the  two  later  time  periods,  all  participants  tended  to  rate  software  support  so  that 
scores  followed  a  more  normal  distribution  pattern  centered  closer  to  the  good  rating  as  shown  by  numerical 
averages.  Two  other  notable  exceptions  exist  to  the  trend  implied  by  overall  score  distributions.  All  PEO/PM 
participants  scored  software  support  between  1998  and  2004  as  a  numerical  average  4  or  good.  After  2004, 
however,  PEO/PM  participants  scored  software  support  with  an  average  of  3.7  between  good  and  adequate.  Other 
participants  scored  software  support  prior  to  1998  and  between  1998  and  2004  with  an  average  of  3.33  and  3.25, 
respectively.  These  lower  scores  contributed  to  reduced  scoring  averages  and  affected  the  distribution  of  scores 
across  the  Army  C4ISR  sample  population. 
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Participants  suggested  areas  in  which  improvements  could  be  made  in  software  support.  Over  all  respondents, 
improving  the  cost  of  field  support  was  the  most  frequent  suggestion  by  a  substantial  margin.  Other  improvement 
suggestions  in  order  of  frequency  included  timeliness  of  accreditation,  timeliness  of  software  upgrades,  improving 
the  quality  and  thoroughness  of  field  support,  improving  the  quality  of  software  upgrades,  and  improving  the 
quality  and  availability  of  online  and  help  desk  support.  Participants  also  provided  a  variety  of  comments,  a 
number  of  which  focused  on  cost  concerns  and  concerns  about  the  capabilities  of  field  support  engineers  and 
technicians. 

This  chapter  described  and  summarized  results  of  the  survey.  These  results  will  be  used  in  the  final  chapter  to 
develop  conclusions  about  the  research  questions.  The  final  chapter  will  consider  study  limitations  and  assumptions 
while  drawing  these  conclusions  and  propose  further  studies  that  may  clarify  or  expand  these  conclusions. 
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CHAPTER  5 

CONCLUSIONS  AND  RECOMMENDATIONS 


Introduction 

This  study  was  intended  to  determine  if  two  decisions  within  the  U.S.  Army  changed  the  perception  of 
software  sustainment  support  for  C4ISR  systems.  The  study  also  considered  the  perceptions  different  sectors  of 
the  Army  C4ISR  community  have  regarding  software  sustainment  support.  As  an  ever-increasing  part  of  system 
performance,  software  requires  support  necessary  to  meet  operational  requirements  so  it  can  perform  when  the 
mission  demands. 

This  chapter  of  the  study  will  review  results  of  the  research  compared  to  research  questions  and  hypotheses, 
review  study  limitations,  and  draw  conclusions.  Implications  for  further  research  and  recommendations  are  also 
presented  followed  by  a  summary  and  conclusion  statement. 

Summary  Statement 

Based  on  the  three  hypotheses,  the  following  are  summary  results  of  the  study. 

(HI)  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of  software 
sustainment  support  for  C4ISR  systems  after  the  1997  ASA(RDA)  decision  compared  to  the  perception  of  C4ISR 
software  sustainment  support  prior  to  1997.  The  study  generally  supports  this  hypothesis.  Survey  results  show  an 
average  software  support  score  of  3.46  (between  good  and  adequate)  prior  to  1998  and  an  average  score  of  3.74 
between  1998  and  2004.  These  averages  and  the  distribution  of  survey  scores  indicate  the  general  perception  of 
software  support  after  the  1997  ASA(RDA)  decision  to  make  PMs  responsible  for  system  life-cycle  support  was 
the  same,  or  only  slightly  better  than  the  perception  of  software  support  after  the  decision. 

(H2)  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of  software 
sustainment  support  for  C4ISR  systems  after  the  2004  ASA(ALT)/AMC  MOA  compared  to  the  perception  of 
C4ISR  software  sustainment  support  prior  to  2004.  The  study  generally  supports  this  hypothesis.  Average  survey 
scores  for  software  support  between  1998  and  2004  were  3.74.  After  2004,  average  survey  scores  were  3.9.  These 
averages  and  the  distribution  of  scores  indicate  the  general  perception  of  software  support  after  the  2004  MOA 
between  the  Army  ASA(ALT)  and  CG  AMC  was  the  same,  or  only  slightly  better. 

(H3)  Among  users,  developers,  and  software  sustainers,  there  is  no  difference  in  the  perception  of  software 
sustainment  support  for  C4ISR  systems  today.  This  hypothesis  is  not  supported  by  the  study.  The  PEO/PM  and 
Other  sectors  of  the  Army  C4ISR  population  sample  tended  to  score  software  support  in  all  three  time  periods  more 
tightly  around  a  single  scoring  value.  The  SSSA  sector  tended  to  score  software  support  with  a  wider  distribution 
across  scoring  levels.  The  central  tendency  for  the  three  sectors  after  2004  was  very  similar.  However,  the  Other 
sector  had  a  central  tendency  notably  closer  to  adequate  than  good  prior  to  1998  and  between  1998  and  2004. 

When  suggesting  areas  of  improvement  for  software  support,  the  SSSA  sector  very  clearly  felt  the  cost  of 
field  software  support  was  primary.  Quality  and  thoroughness  of  field  support  and  quality  and  availability  of  help 
desk  support  were  a  distant  second  for  SSSA  respondents,  while  quality  and  availability  of  online  support  and 
timeliness  of  system  accreditation  tied  for  third.  PEO/PM  participants  believed  timeliness  of  system  accreditation 
was  primary,  with  timeliness  of  software  upgrades  and  cost  of  field  support  upgrades  second  and  third,  respectively. 

Based  on  the  results,  the  original  study  questions  can  be  answered  within  the  limits  and  assumptions  of  the 
research. 

(1)  Is  the  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among  users/units,  PEOs/PMs 
and  software  support  service  providers  better  or  worse  since  the  1997  ASA(RDA)  decision? 

(2)  Is  the  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among  users/units,  PEOs/ 
PMs,  and  software  support  service  providers  better  or  worse  since  the  2004  MOA  between  the  AS  A(ALT)  and  CG 
AMC? 

(3)  What  is  the  current  perception  of  software  sustainment  support  for  Army  C4ISR  systems  among  users/ 
units,  PEOs/PMs,  and  software  support  service  providers? 
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The  perception  of  software  sustainment  support  is  generally  the  same  or  only  slightly  better  among  the  PEO/ 
PM  and  SSSA  sectors  of  the  Army  C4ISR  communities  since  the  1997  and  2004  Army  policy  decisions  were  made. 
While  survey  participants  agree  that  improvements  should  be  made,  the  distribution  of  suggested  improvements 
differed  substantially  among  the  C4ISR  community  sectors. 

Limitations 

Chapter  1  explained  the  limited  nature  of  the  Army  C4ISR  community  and  described  the  inability  to  achieve 
a  statistically  significant  sample  size  of  this  community  due  to  the  limited  response  to  the  survey.  The  study 
was  also  confined  to  perceptions  of  members  of  the  Army  C4ISR  community.  It  did  not  address  objective  data 
regarding  the  quality,  timeliness,  cost,  and  thoroughness  of  software  sustainment  support.  These  limitations  will 
contribute  to  suggestions  for  further  and  expanded  research  in  this  study  area. 

Implications  for  Further  Research 

This  research  broadly  demonstrates  that  studies  can  be  performed  which  provide  information  of  value  to  DoD 
leaders  in  assessing  the  impacts  of  their  acquisition  policy  decisions.  Conducting  these  studies  can  also  assist  in 
formulating  future  policy  changes.  While  this  study  was  limited  to  the  perception  of  participants,  studies  should 
also  be  conducted  on  objective  data  regarding  software  support  quality,  timeliness,  cost,  and  thoroughness.  When 
merged  with  data  on  participant  perceptions,  a  more  complete  picture  of  software  sustainment  support  can  be 
provided  to  leaders.  This  type  of  research  can  be  expanded  beyond  the  field  of  software  support  within  the  Army 
C4ISR  community  to  many  different  acquisition  areas  within  DoD.  Research  limitations  point  to  the  need  for 
better  mechanisms  to  distribute  surveys  and  other  research  vehicles  in  order  to  improve  the  statistical  significance, 
application,  and  scope  of  research.  Rather  than  rely  on  key  participants  to  further  distribute  research  vehicles, 
studies  should  directly  contact  the  entire  community.  While  not  all  members  will  respond,  contacting  all  members 
will  increase  the  likelihood  and  number  of  responses,  thus  increasing  the  sample  size  and  improving  the  statistical 
significance  of  study  results. 


SUMMARY  AND  CONCLUSIONS 

This  study  was  limited  to  the  sustainment  support  of  software  within  Army  C4ISR  systems.  A  survey  of  a 
small  sample  of  the  Army  C4ISR  community  provided  information  on  the  perception  of  software  sustainment 
support  before  and  after  two  significant  acquisition  policy  decisions.  One  decision  in  1997  by  the  ASA(RDA) 
made  PMs  responsible  for  the  entire  life-cycle  management  of  a  program.  In  2004,  the  ASA(ALT)  and  CG  AMC 
signed  an  MOA  creating  LCMCs  and  more  closely  associating  these  organizations  with  PEOs. 

Results  indicate  the  perception  of  software  sustainment  support  for  Army  C4ISR  systems  has,  in  general, 
remained  the  same  or  only  slightly  improved  during  each  of  the  three  periods  in  question,  supporting  the  first  two 
study  hypotheses.  However,  the  varied  scoring  among  different  sectors  of  the  Army  C4ISR  population  sample 
indicate  the  sustainment  support  community  may  have  perceptions  that  are  more  widely  distributed,  even  if  the 
central  tendency  is  similar  to  other  sectors.  Results  did  not  support  the  hypothesis  that  different  subgroups  of  the 
Army  C4ISR  community  have  the  same  perceptions  about  software  sustainment  support.  Instead,  results  indicate 
sectors  of  the  Army  C4ISR  community  differ  regarding  what  areas  of  software  sustainment  support  need  further 
improvement.  While  most  PEO/PM  personnel  appear  to  be  concerned  about  system  accreditation,  most  SSSA 
personnel  indicate  concern  with  the  cost  of  field  software  support. 

The  study  indicates  there  is  value  to  conducting  similar  and  expanded  research  on  various  acquisition  processes, 
the  perception  of  those  processes,  and  the  impact  of  policy  decisions  on  those  processes.  If  these  additional  and 
expanded  studies  overcome  study  limitations,  analysis  resulting  from  data  collected  will  be  more  useful  and 
valid  across  broader  aspects  of  the  acquisition  enterprise.  For  example,  there  are  at  least  two  interpretations  of 
these  study  results.  A  positive  perspective  would  be  that  the  perception  of  software  support  following  the  two 
acquisition  decisions  remained  the  same,  or  improved  slightly.  A  less  positive  perspective  could  be  that  even  after 
expending  significant  resources  and  effort  to  implement  the  two  acquisition  decisions,  the  perception  of  software 
support  for  Army  C4ISR  systems  did  not  significantly  improve.  Is  one  perspective  correct?  Are  both  perspectives 
correct?  This  study  cannot  answer  these  questions. 
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The  study  does  point  to  the  possibility  that  significant  acquisition  policy  changes,  and  the  investment  required 
to  implement  those  changes,  may  not  result  in  equivalent  benefits.  This  suggests  that  a  more  robust  system  able  to 
measure  the  cost  savings  and  effectiveness  of  major  acquisition  policy  changes  may  be  needed  to  inform  decisions 
made  by  our  senior  leaders. 

The  business  of  military  system  acquisition  doesn’t  begin  with  a  contract  request  for  proposal  and  end 
with  the  fielding  of  a  product.  System  acquisition  begins  with  a  statement  of  need.  That  need  will  not  be 
met  with  the  delivery  of  a  system.  The  need  is  met  with  an  operational  system,  ready  to  perform  when  the 
mission  demands.  Throughout  its  life,  the  system  requires  support  both  to  maintain  existing  functionality,  and 
to  enhance  functionality  to  meet  changing  environments  and  threats.  As  business  and  geopolitical  conditions 
change,  DoD  acquisition  leaders  develop,  review,  and  modify  acquisition  policies,  programs,  and  processes  to 
provide  warfighters  the  systems  they  need  to  win.  In  the  current  return  to  fiscal  austerity,  DoD  leaders  require 
robust  information  to  inform  decisions  and  ensure  that  the  most  cost-efficient  and  mission-effective  acquisition 
policies  are  implemented.  Studies  such  as  this  one  can  contribute  information  that  helps  leaders  develop  and 
execute  acquisition  policies  to  meet  warfighter  needs. 
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GLOSSARY  OF  ACRONYMS  AND  TERMS 


AKO 

Army  Knowledge  Online 

AMC 

Army  Materiel  Command 

AMCOM 

Aviation  and  Missile  Command 

AR 

Army  Regulation 

ASA(ALT) 

Assistant  Secretary  of  the  Army  for  Acquisition,  Logistics,  and  Technology 

ASA(RDA) 

Assistant  Secretary  of  the  Army  for  Research,  Development,  and  Acquisition 

C4ISR 

Command,  Control,  Communications,  Computers,  Intelligence,  Surveillance,  and 
Reconnaissance 

CECOM 

Communications-Electronics  Command 

CG 

Commanding  General 

DoD 

Department  of  Defense 

LCMC 

Life  Cycle  Management  Command 

MOA 

Memorandum  of  Agreement 

MSC 

Major  Subordinate  Command 

PEO 

Program  Executive  Office 

PEO  C3T 

PEO  Command,  Control,  Communications,  and  Technology 

PEO  EIS 

PEO  Enterprise  Information  Systems 

PEO  IEW&S  PEO  Intelligence,  Electronic  Warfare  and  Surveillance 


PM 

Program  Manager 

PMO 

Program  Management  Office 

STRICOM 

Simulation  and  Training  Technology  Command 

TACOM 

Tank  and  Automotive  Command 

URL 

Uniform  Resource  Locator 

USD(AT&L)  Under  Secretary  of  Defense  for  Acquisition  Technology  and  Logistics 
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HELLO, 


I  am  conducting  research  on  the  perception  of  software  sustainment  support  for  Army  C4ISR  systems.  Survey 
results  will  be  published  in  a  research  paper  for  the  Defense  Acquisition  University  (DAU)  with  the  potential  for 
wide  distribution  within  the  Army  acquisition  workforce.  Copies  of  this  research  paper  will  be  available  upon 
request  pending  publication  and  approval  by  DAU. 

Names  of  survey  participants  will  not  be  revealed  in  the  research  paper.  During  a  pilot  survey,  participants 
took  less  than  10  minutes  to  complete  the  questionnaire.  Please  select  the  best  response  to  each  question.  If  you 
know  of  other  Army  C4ISR  system  PEO/PM,  software  support,  or  user/operator  personnel  who  should  complete 
this  survey,  request  you  provide  them  this  Wwebsite. 

I  greatly  appreciate  your  participation  in  the  survey. 

Steven  C.  Cooper 
Fellow 

Defense  Acquisition  University 
Aberdeen  Proving  Ground,  MD 


1 .  I  am  currently  an  Army  C4ISR,  or  am  affiliated  with  an  Army  C4ISR  (Please  check  only  one  item  that  best 
applies): 

[  ]  System  User/Operator 
[  ]  System  PEO/PM 
[  ]  System  Software  Support  Activity 
[  ]  System  Development/Support/User  Contractor 
[  ]  Other 

2.  I  have  been  affiliated  with  the  operation,  development,  and/or  support  of  Army  C4ISR  systems  (Please  check 
only  one  item  that  best  applies): 

[  ]  Since  Before  1998 

[  ]  Since  Before  2004  But  After  1998 

[  ]  Since  2004 
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3.  Prior  to  1998  software  support  for  Army  C4ISR  systems  after  fielding  by  the  PM  was  (Please  check  only 
one): 

If  you  checked  “Since  Before  2004  But  After  1998”  in  question  2,  please  skip  this  question  and  go  to  questio  4. 
If  you  checked  “Since  2004”  in  question  2,  please  skip  this  question  and  go  to  question  5. 

[  ]  Excellent 

[  ]  Good 

[  ]  Adequate 

[  ]  Poor 

[  ]  Very  Poor 

4.  Between  1998  and  2004  software  support  to  C4ISR  systems  after  fielding  by  the  PM  was  (Please  check  only 
one): 

If  you  checked  “Since  2004”  in  item  2,  please  skip  this  question  and  go  to  question  5. 

[  ]  Excellent 
[  ]  Good 
[  ]  Adequate 
[  ]  Poor 
[  ]  Very  Poor 

5.  Overall,  current  software  support  to  C4ISR  systems  after  delivery  by  the  PM  has  been  (Please  check  only 
one): 

[  ]  Excellent 
[  ]  Good 
[  ]  Adequate 
[  ]  Poor 
[  ]  Very  Poor 
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6.  Please  select  the  areas  you  believe  require  improvement  in  providing  software  support  (Please  check  all  that 
apply): 

[  ]  Quality  of  Software  Upgrades 

[  ]  Cost  of  Software  Upgrades 

[  ]  Timeliness  of  Software  Upgrades 

[  ]  Quality/Thoroughness  of  Field  Support 

[  ]  Cost  of  Field  Support 

[  ]  Timeliness  of  Field  Support 

[  ]  Quality/Availability  of  On-line  Support 

[  ]  Cost  of  Online  Support 

[  ]  Quality/ Availability  of  Help  Desk 

[  ]  Cost  of  Help  Desk 

[  ]  Quality  of  Information  Assurance 

[  ]  Cost  of  Infonnation  Assurance 

[  ]  Implementation  of  Information  Assurance 

[  ]  Quality  of  System  Accreditation 

[  ]  Cost  of  System  Accreditation 

[  ]  Timeliness  of  System  Accreditation 

[  ]  Other 

7.  Please  provide  any  comments  you  may  have  regarding  software  sustainment  support  for  Anny  C4ISR 
systems: 
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RECORD 

NUMBER 

COMMENTS 

J0002 

Many  times  software  sustainment  by  organic  capability  is  not  consider  by  ASAA- 
LT  community,  nor  previously  stressed  by  DA  in  general.  .  .  .  Pushing  QRCs  out 
by  PM  offices  is  one  thing,  but  not  doing  due  diligence  in  terms  of  procuring  data 
rights  and  planning  for  transition  from  ORM  has  been  a  large  problem  putting  the 
Army  at  risk  and  cost. 

J0003 

Over  the  period  collected  by  the  survey  field  support  of  software  for  C4ISR 
systems  has  grown  tremendously  in  response  to  multiple  factors  including  sys¬ 
tem  complexity,  requirements  for  readiness,  contingency  operation  requirements, 
etc.  The  growth  of  field  support  personnel  was  required  to  meet  this  requirement. 
Initial  field  support  to  C4ISR  systems  had  top  engineers  that  were  experienced 
with  the  systems  during  their  initial  development,  this  is  no  longer  the  case  so  the 
experience  level  of  the  FSE  has  degraded  over  the  entire  period. 

J0006 

Software  Sustainment  should  a  key  parameter  in  the  design  and  development  of 
the  system.  The  U.S.  Army  should  have  a  software  sustainment  plan  and  process 
for  each  system. 

J0007 

I  believe  the  majority  of  staffers  don't  have  a  clue  as  to  what's  involved/required  to 
sustain  software.  There  appears  to  be  a  belief  that  sustainment  is  simply  keeping 
the  software  running  as  long  as  possible.  However,  software  sustainment  today 
really  incorporates  software  changes  as  the  business  changes,  incorporating  new 
technologies  to  replace  those  that  are  no  longer  supported.  It’s  adapting  that  soft¬ 
ware  to  new  environments  -  either  new  hardware,  operating  systems  or  network 
connections.  It's  keeping  the  entire  "system"  current  and  useful  and  extending  its 
life  cycle  utilization. 

F0002 

Software  Sustainment  needs  to  address  remediation  in  systems  in  sustainment  of 
critical  software  assurance  deficiencies  that  are  defined  as  critical  in  accordance 
with  the  Application  Security  Technical  Implementation  Guide  of  the  Defense 
Information  Systems  Agency  as  noted  in  Section  932  related  to  Information  As¬ 
surance  policy  as  part  of  the  DoD  Authorization  Bill  FY201 1.  These  deficien¬ 
cies  will  not  only  incur  security  risk  but  will  increase  costs  and  chance  of  system 
failure  as  the  system  is  maintained  in  the  out  years. 

F0003 

Many  PMs  do  not  transition  their  programs  to  PDSS/PPSS  in  an  expected  time- 
frame.  Transitioning  your  system  as  early  as  practical  will  free  up  resources  at 
the  PM  level  and  also  enable  sustainment  funding. 

F0004 

Generals  and  high  officials  get  involved  in  problems  that  can  be  easily  resolved 
by  developers.  Lots  of  government  expertise  has  driven  up  contractor  cost.  Aca¬ 
demia  and  PHD  type  folks  have  driven  away  common  sense  and  stick-to-it  men¬ 
tality.  We  make  fewer  mistakes,  but  leam  nothing.  The  process  works,  but  who 
can  afford  it?  I  think  that  the  acquisition  strategy  is  designed  to  deliver  the  best 
product,  but  at  a  cost  the  government  cannot  afford  to  pay. 

F0005 

Coordination  between  dependent  system  and  meeting  schedules. 

F0008 

I  do  not  have  any  knowledge  of  software  sustainment  support. 
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F0012 

Timeliness  is  a  big  thing  for  supporting  the  systems.  One  of  the  items  that  con¬ 
tributes  to  long  delays  is  the  configuration  management  process.  Configuration 
management  is  extremely  necessary,  but  some  way  needs  to  be  found  to  stream¬ 
line  the  process  and  make  software  updates,  upgrades  and  over-all  support  more 
timely. 

F0014 

Need  FSRs  who  are  capable  of  maintaining  multiple  systems  instead  of  FSRs 
who  are  specialized  in  only  one  system. 

F0019 

Funding  process  and  commitment  of  same  by  SEC  leaves  much  to  be  desired. 
System  is  very  broken  and  PM  is  left  to  fight  for  every  penny.  Special  problem 
areas  are  sustainment  funding  after  transition  and  license  maintenance.  Typical 
SEC  scenario  is  to  "lowball"  everything  and  create  much  extra  work  for  the  PM 
staff  to  justify  and  rejustify  to  the  extreme.  I  have  not  seen  the  "learn"  effort  that 
should  be  present  in  this  process.  It's  "US"  vs  "THEM"  and  that  is  definitely  NOT 
the  way  the  Army  intended.  Other  areas  of  support  (especially  FSE  and  Informa¬ 
tion  Assurance  have  been  EXCELLENT. 

F0020 

COCOMs  have  sufficient  funding  to  build  their  own  stovepiped  systems.  These 
efforts  have  a  direct  adverse  effect  on  the  use,  maintenance,  and  effectiveness  of 
system  of  record  C4ISR  systems. 

F0022 

Too  disjointed.  We  fail  to  recognize  the  difference  between  contract  management 
and  operational  support.  Current  system  geared  more  to  the  management  piece 
vice  problem  identification,  management,  tracking  and  resolution. 

F0023 

System  support  needs  to  be  adopted  in  a  manner  that  recognizes  that  Depot 
and  Field  Support  cannot  work  together  if  leadership  of  same  is  separate.  Hav¬ 
ing  been  associated  with  software  programs  for  more  than  10  years  now,  I  have 
witnessed  a  fairly  abrupt  change  to  how  Field  Support  responds  to  customer  re¬ 
quirements  and  in  some  cases  a  perceived  indifference  to  Depot  level  support.  It 
makes  it  very  complex  for  an  end  user  to  understand  who  is  providing  what  level 
of  support  to  their  problem.  I  don't  quite  understand  how  this  then  translates  to 
Single  Face  to  the  Field. 

F0024 

Make  IA  and  SW  patch  updates  easier  for  units  to  handle  on  their  own  instead 
of  relying  on  contractor  support.  This  becomes  an  issue  when  the  lower-density 
systems  are  due  to  receive  a  patch  yet  the  FSRs  are  not  readily  available  to  assist 
due  to  their  low  numbers  and  prior  commitments. 

F0025 

As  software  becomes  more  complex,  we  are  more  reliant  on  contractor  support 
vice  "soldier"  support.  In  the  late  eighties  through  late  nineties,  most  trouble¬ 
shooting  and  basic  fixes  were  done  by  the  soldier/operator  (how  I  became  in¬ 
volved  and  interested  in  support.)  Even  more  complex  fixes  were  done  by  the  sol¬ 
dier/operator  with  telephonic/mail  assistance.  Now,  even  basic  trouble  shooting 
cannot  be  conducted  by  the  soldier/operator;  it  requires  a  Field  Support  contractor 
(or  System  Administrator)  to  evaluate  problems  beyond  "it's  not  working  right" 
putting  an  enormous  strain  on  the  limited  workforce. 
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F0026 

The  Unit  Set  Fielding  process  by  which  the  units  are  given  the  training  and  then 
fielded  the  systems  in  a  coordinated  and  synchronized  manner  has  helped  the 
units  better  understand  the  network  they  are  operating,  rather  than  just  a  few  of 
the  individual  systems.  The  RESET  process  does  the  same  for  redeploying  units. 
The  digital  system  engineer  program  has  helped  sync  the  field  support  to  the 

C4ISR  systems  by  the  FSRs,  especially  in  the  units  with  many  digital  systems. 

The  training  and  experience  within  the  field  workforce  has  increased  which  has 
enabled  the  units  to  better  utilize  the  systems. 

F0027 

I  believe  we  have  come  a  long  ways  in  providing  sustainment  support  over  the 
years,  but  we  have  become  prisoners  to  the  costs  of  contractors.  We  need  to  go 
back  to  the  soldiers  being  skilled  on  their  own  C4ISR  systems  to  minimize  the 
need  for  over  charging  contractors. 

F0028 

The  lack  of  qualified  FSEs  that  have  the  capability  to  support  any  of  the  PM 
platforms  that  falls  under  the  contract.  Having  system  specific  FSEs  hampers  the 
ability  to  deploy  in  support  of  multi-mission  requirements.  That  makes  for  higher 
unit  costs  because  they  have  to  take  more  contractors. 

F0029 

Availability  of  FSE  personnel  is  limited  due  to  budgetary  constraints  and  lack  of 
trained  personnel.  Although  previously  deleted,  the  assignment  of  FSE  directly  to 
units  is  a  must  for  successful  operations.  Solid  FSE  and  management  are  over¬ 
coming  this  constraint,  but  the  support  force  is  stretched  thin. 

M0002 

SW  is  viewed  as  an  anomaly  vs  HW.  Feadership  doesn't  understand  SW  sustain¬ 
ment  maximizes  a  system  investment  ROI  (being  too  quick  to  suggest  letting 
fielded  systems  obsolesce,  then  need  risky  expensive  replacements.)  These  same 
people  wouldn't  suggest  buying  a  fleet  of  tanks/trucks  and  not  providing  for  their 
maintenance,  yet  do  just  that  for  their  SW.  Uneven  approach  to  SW  sustainment 
leaves  to  each  PM  to  decide,  but  should  be  done  in  institutionalized  governance 
process  that  considers  the  entire  portfolio  and  lifecycle.  Uneven  field  support 
should  be  institutionalized  via  AMC  in  support  of  PM. 
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